test: increasing unit test coverage to 95 percent - #740
Conversation
|
Hi @mgama-deepgram this is great, however some of the files you've modified are generated files based on the API specs. I'd rather we didn't freeze files unless absolutely unavoidable. |
|
@GregHolmes I removed all the changes that were done to files under src/deepgram/core and left only the new tests under tests/custom. |
@mgama-deepgram would you not want your other test files in .fernignore? otherwise when regenerations done, it'll remove them. |
Thanks for calling this out @GregHolmes I just added the tests files to the .fernignore file. I am new to use these type of files and regenerations, thanks for explaining. |
|
@GregHolmes this is ready for another round of review. I added more test coverage for Speak V2 (Flux TTS). Thanks in advance |
|
|
@dg-coreylweathers @jherl-dg this is ready for review. I appreciate your help, thanks in advance |
|
@dg-coreylweathers @GregHolmes these changes are ready to be re-review. I have addressed the original changes requested from the first review. Thanks in advance |
45010dc to
0f3ed64
Compare
Removes the committed .coverage binary from version control. It is a local coverage artifact regenerated on every test run and is already listed in .gitignore.
Pins irongut/CodeCoverageSummary (v1.3.0) and marocchino/sticky-pull-request-comment (v2.9.4) to commit SHAs. Both run with pull-requests: write, so pinning to an immutable SHA (rather than a moving tag) hardens the privileged coverage-comment steps.
CodeCoverageSummary is a Docker container action that writes code-coverage-results.md as root, so the follow-up annotate step's plain `>>` append ran as the runner user and failed with 'Permission denied', failing the 3.13 leg and cancelling the rest of the matrix (fail-fast). Append as root via 'sudo tee -a' instead.
Increase unit test coverage (~95%) + coverage in CI
Adds a hand-written custom test suite that raises
deepgrampackage coverage to the ~95% target (branch coverage enabled), wires coverage reporting into CI with an automatic sticky PR comment, and scopes coverage to hand-maintainable logic rather than Fern-generated boilerplate. New tests and the coverage config are frozen in.fernignoreso they survive the next SDK regeneration.Coverage on the scoped source went from ~54% line / ~32% branch to 94.72% (gate
fail_under = 90). No product code changed — this PR is tests + CI + config only.What changed
CI (
.github/workflows/ci.yml)pull_requestin addition topush(push now scoped tomain+ tags).pytest -rP --cov=deepgram --cov-branch --cov-report=xml --cov-report=term-missing .pull-requests: write(withcontents: read) so it can comment.Coverage config (
.coveragerc, notpyproject.toml)[run]branch coverage on,source = deepgram, with pure-generated code excluded (types/,requests/,__init__.py,version.py, unusedcore/http_sse/) so the metric reflects code that carries logic.[report] fail_under = 90,show_missing = True.pyproject.tomldeliberately: freezingpyproject.tomlwould also block Fern's generator dependency updates.Dev dependencies (
pyproject.toml)pytest-covandrespx(HTTP mocking) to the dev group.New tests (
tests/custom/, ~170 test functions, many parametrized)test_http_endpoints_coverage.py— table-driven sweep of REST endpoints (success + 400/403/non-JSON error branches, sync + async).test_http_retry_coverage.py— retry/backoff behavior (exponential backoff,Retry-After,Retry-After-ms,X-RateLimit-Reset, exhaustion).test_websocket_streaming_coverage.py— websocket streaming clients.test_speak_v2_coverage.py— Speak V2 (Flux TTS): batch RESTaudio.generatestreaming, connect header/handshake-failure branches, socket-client iteration/send/recv. Raisesspeak.v2from 44% → 97%.test_core_internals_branches.py,test_core_utilities_coverage.py,test_client_construction_and_helpers.py,test_logging_and_retry_branches.py— core utilities, client construction, logging/retry, and remaining branch gaps.Regen safety / housekeeping
.fernignore— freezes the 8 new custom test files,.coveragerc, and.gitignore. Documented inAGENTS.md..gitignore— ignores.coverage,htmlcov/,coverage.xml(and the previously-tracked.coverageartifact is now untracked).main(7.7.0 regen) to keep the branch current.Testing
respx(HTTP) and in-process websocket fakes — no network / no WireMock dependency.981 passed, 1 skippedunder coverage locally; CI runs the full suite on Python 3.10–3.13.Open item (for @greg / Fern pipeline):
pyproject.tomlis intentionally unfrozen, but now carries therespx/pytest-covdev deps the frozen tests need. Confirm these survive the next Fernpyproject.tomlregen (or move them into the generator's dependency config) so the frozen suite doesn't break on a future regen.